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Introduction 


This report provides a re-validation of the Florida Department of Corrections (DOC) 
Correctional Operations Trend Analysis System (COTAS) by two reviewers, one from Florida 
State University’s (FSU) College of Criminology and Criminal Justice’s Center for Criminology 
and Public Policy and one from Infinity Software. COTAS is designed to serve as a tool for DOC 
staff and administrators in “identifying issues before they develop ” by consolidating historical 
information with a predictive scoring system made readily available on a comprehensive 
website. COTAS provides DOC staff and administrators with two types of statistics, descriptive 
and predictive. Descriptive statistics from COTAS provide a summary of violent and non-violent 
events that occurred within DOC regions and facilities within the prior 30 days. This data can be 
“drilled down” to examine the prevalence of events at the facility, dorm, and inmate levels. 
Predictive statistics from COTAS provide the predicted probability of individual inmates’ 
involvement in violent events. Predictions are generated by an algorithm, which uses historical 
data to examine the relationship between inmate and facility characteristics and inmates’ 
involvement in violent events. Both descriptive and predictive statistics are reported to the user 
in a web-based dashboard interface. 

COTAS was validated for the first time between February 201 1 and May 201 l(final 
report submitted June 201 1) by FSU’s Center for Criminology and Public Policy and ELENC, 
Inc. This original validation provided the DOC with a list of 21 recommendations with the intent 
of increasing the validity of the predictive models within COTAS, the understanding of the 
utility and function of COTAS from the perspective of the users (DOC staff and administrators), 
and the validity of the COTAS software. The reviewers from FSU and Infinity Software were 
tasked with examining the 21 recommendations provided by the original validation of COTAS 
and determining whether or not each recommendation was addressed and if so, the extent to 
which it was addressed or implemented by the DOC. 

Overview of the Original Validation 

The original validation of COTAS was a joint effort by FSU’s Center for Criminology 
and Public Policy and ELENC, Inc. The reviewers from FSU conducted a validation of the 
predictive models within COTAS and the understanding of the utility and function of COTAS 
from the perspective of COTAS users (DOC staff and administrators), and the reviewers from 
ELENC, Inc. conducted a validation of the COTAS software. 

Predictive Models 

This portion of the validation was concerned with the predictive models used to predict 
which inmates and facilities were most likely to experience a violent event. In the original 
validation the reviewers concluded that the predictive models within COTAS provided 
correctional administrators with useful information regarding the prevalence of violent and non- 
violent events within DOC facilities. However, the models that predicted inmates’ involvement 
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in violent events, the documentation of the software ETL procedures and the tests of data 
accuracy were areas that warranted attention from the DOC to improve the overall accuracy and 
functionality of COTAS. One of the most pressing concerns was the need to create a codebook 
or data dictionary for all of the data used by COTAS. Additionally, the models of future 
involvement in violent events had low predictive accuracy and could be improved by 
restructuring the data sets to include the time ordering of the events. This resulted in a total of 7 
recommendations (recommendations #4-8 & 10-12). 

User’s experience with COTAS 

In this portion of the validation, the reviewers were tasked with assessing the utility, 
function, and ease of use of COTAS from the perspective of the DOC staff and administrators 
who use it on a day to day basis. To accomplish this task, a user survey was created to gain an 
understanding of the extent to which correctional staff and administrators use COTAS, and more 
specifically, which components of COTAS are most useful. Prior to the original validation, the 
DOC had developed and administered a pilot survey with similar intentions. However the results 
were inconclusive because the survey instrument was limited in scope and the response rate was 
very low (n=2). The results of the survey indicated that a significant portion (42%) of DOC staff 
and administrators who have access to COTAS do not use this tool. Of those who use COTAS, 
they use it on a relatively frequent basis. The most significant concern from the users’ 
perspective was the inability to understand the distinctions between the color coding of facilities 
(red, yellow, and green) as well as the differences between the screens. At the time of the 
original validation, a user’s manual was being created and had not yet been disseminated to the 
COTAS users. Additionally 90% of COTAS users agreed that COTAS would be improved with 
an updated user’s manual. The reviewers concluded that once the updated user’s manual was 
disseminated, the majority of the confusion about the color coding and the different screens 
within COTAS would be addressed. As a result of this validation, 6 recommendations 
(recommendations #1-3, 9, 13, & 14) were suggested for improvement of the user’s experience 
with the COTAS software. 

Software Validation 

This purpose of this portion of the validation was to validate the technical aspects of the 
software itself - how it was constructed, how it was implemented, how it was documented, and 
the ability of the software to transferred to other agencies. The construction and implementation 
of the software was validated through examination of the documentation and a detailed analysis 
of the actual working code and data structures. The reviewers attempted to compare the 
documentation and code with accepted industry practices. The original review discovered 
deficiencies in the design of the data warehouse itself (Recommendation 18) as well as the data 
routines used to load data from source systems into the warehouse (Recommendation 19). The 
documentation of the software was examined and found to be deficient (Recommendations 17 
and 20). It was determined by the reviewers that the transferability of the software was primarily 
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impacted by the lack of documentation and made no specific recommendations on that portion of 
the review although the final recommendation concerning the staff used may be taken as a 
recommendation for any future work - including transfer of the software to another agency 
(Recommendation 21). 


Purpose of the Re- Validation 

While COTAS provides access to valuable data and information for correctional 
administrators and staff, there were deficiencies to the predictive models, documentation, and 
software that negatively impacted: (1) the degree of accuracy in predicting the likelihood of 
violence in the institutions, (2) the ease with which the information is communicated and 
understood by users, and (3) the ability for COTAS to be replicated in other correctional settings. 
Therefore, at the conclusion of the original validation, 21 recommendations were suggested to 
improve the overall effectiveness of COTAS. Thus, outside reviewers are necessary to determine 
whether, and to what extent, the recommendations have been addressed. 

Methodology of the Re- Validation 

FSU’s College of Criminology and Criminal Justice, Center for Criminology and Public 
Policy Research worked together with Infinity Software to validate the 21 recommendations 
provided by the original validation. Leslie Hill, from FSU validated recommendations #1-14, 
while Richard Peterson, from Infinity Software validated recommendations #15-21. Each 
reviewer had different tasks to address, thus their methodologies were slightly different. Each 
reviewer’s methodology will be discussed in the following section. 

FSU Center for Criminology and Public Policy Research: Recommendations #1-14 

Several of the recommendations (1, 3, 8 & 9) required the updated user’s manual. As 
explained above, the majority of the user’s confusion regarding COTAS would be quelled with 
an updated user’s manual. The user’s manual is available on the COTAS main page. To validate 
recommendations #1, 3, 8 & 9, the reviewer thoroughly read through the user’s manual and took 
notes where there were any areas of potential confusion. These were then discussed with Marien 
Dimacali, a contractor for e-systems database, and Gail Denson, the Project Manager of the 
DOC’s Office of Information Technology. Several of the other recommendations (2, 3, 6, 7, 12, 
13, & 14) also required consulting with Gail and Marien. The reviewer went out to the DOC on 
several occasions to meet with Gail and Marien to go over recommendations or issues found 
within COTAS. For example, to validate recommendation #2, extensive consultation with Gail 
was required to determine the extent to which training opportunities had been provided to 
COTAS users. Additionally, email exchanges worked very well to quickly answer a question or 
to provide examples of problems within COTAS for Marien or Gail to research further. For 
example, recommendation #10 involved determining whether or not the RegionGraph and the 
EventGraph were rate or incident counts. Marien was emailed the question and she was able to 
go back through the code and respond to the email with the answer. The remainder of the 
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recommendations required a thorough examination of the different screens within COTAS. The 
reviewer systematically went through each screen of COTAS in which there were recommended 
changes. For example, recommendation #5 indicated that the columns on the side of the 
RegionDashboard did not add up correctly. Therefore the researcher went through the 
RegionDashboards and ensured that the columns were adding up properly. In summary the 
methodology used included a combination of a thorough examination of the COTAS screens, the 
user’s manual, and numerous discussions (in person, email or telephone) with Gail and Marien to 
either clarify something or to determine if a specific recommendation had been addressed. 

Infinity Software: Recommendations #15-21 

The software re-validation was conducted within the strict scope of the previous findings. 
Each finding was reviewed and the necessary documentation or code requested from DOC. In 
some cases, interviews with Gail and Marien were used to determine what steps had been taken 
to address the finding - or in the case of Recommendation 16, David Ensley was asked to 
provide an explanation as to why the recommendation was not implemented. Consistent with the 
initial validation of COTAS, documentation and computer code were reviewed and assessed in 
the light of industry-standard best practices. 


Findings 

The Correctional Operations Trend Analysis System (COTAS) has been validated 
through a joint effort by Florida State University’s (FSU) College of Criminology and Criminal 
Justice, Center for Criminology and Criminal Justice and Infinity Software. Reviewers from FSU 
and Infinity Software went through the 21 recommendations listed in the original validation and 
determined whether or not and to what extent the recommendation was implemented. Therefore 
each recommendation has been validated, meaning the reviewers thoroughly assessed and 
evaluated the extent to which the DOC has addressed and implemented each recommendation. 
The Florida Department of Corrections (DOC) has addressed the vast majority of the issues 
identified in the original validation, conducted in 2011. Although a few small issues remain, 
COTAS is a significantly better system than it was when the original validation study was 
completed. An updated user’s manual has been created and disseminated, which eliminated a lot 
of confusion surrounding COTAS during the original validation. Additionally, the DOC has 
worked to fix almost all of the issues pointed out in the original validation. This report 
documents each of the recommendations provided to the DOC from the original validation and 
determines if the DOC has either addressed or not addressed the recommendations as well as 
document any problems that still exist. Lastly, a summary of the validation will be provided. 

Recommendation #1; Many of the problems associated with the user’s experience can be 
addressed by an updated, expanded user’s manual that is comprehensively disseminated to 
correctional institutions throughout the state. The manual needs to provide complete 
explanations for the establishment of the thresholds (red, yellow, and green), the rationale 


4 


This document is a research report submitted to the U.S. Department of Justice. This report has not 
been published by the Department. Opinions or points of view expressed are those of the author(s) 
and do not necessarily reflect the official position or policies of the U.S. Department of Justice. 



for the thresholds, and the rational for their significance. Also, the manual should clearly 
explain the distinctions for ranking facilities and, in general, provide a more clear and 
comprehensive explanation for COTAS’ functions. DOC currently has two user’s manuals: 
an initial manual and a not yet disseminated revised manual. However, the revised manual 
should be updated based on findings from this validation and pilot tested in the field. 

The DOC has addressed this recommendation by providing a 28 page user’s manual that 
is available on the COTAS main page. The first page provides an overview of COTAS, including 
the primary purpose of COTAS: 

“to identify issues before they develop ” by consolidating historical information 
with a predictive scoring system made readily available through “ dashboards ” 
and “ reports . ’’ 

The introduction explains that two types (referred to as “paths” the users can take from 
the COTAS main page) of information are presented in COTAS: historical and predictive. The 
user’s manual provides an overview and tutorial of each type/path (historical and predictive). It 
makes use of screen shots to help the users understand the corresponding explanation. 
Additionally, the user’s manual provides an overview of the COTAS system based on the 
specific job titles within the DOC (chief security officer, warden, operational staff, etc). As was 
suggested by the original validation, the user’s manual provides an explanation of the differences 
between the color schemes: green: all systems go, yellow: caution, and red: stop and take notice. 
Furthermore, the distinctions between the three colors are thoroughly explained for ranking 
facilities as well as inmate risk scores. 

In addition to the updated user’s manual, a “Threshold Color Code” tab is now available at 
the bottom of the RegionDashboard, which is the welcome page of COTAS. This provides the 
user quick and easy access to the color coding thresholds guide, without being required to close 
out of COTAS and open the user’s manual. This threshold color code tab provides color coding 
explanations and/or formulas for the following screens: 

“Last Month’s Facilities with Violent Events Exceeding Threshold” and “Future 
Predicted Violent Events Based on Inmates” (both from the RegionGauges screen) 

The individual region predictors, both inmate and facility (both from the 
RegionPredictorDashboard screen) 

The percentages that place both the facility and inmate into a specific color (red, yellow, and 
green) are organized into a table for easy viewing. It explains the different percentages for 
different facility types (work release center, institution, re-entry center, etc). During the original 
validation, the DOC staff indicated that there was considerable confusion concerning the 
understanding of the color schemes; therefore the threshold color code tab is an effective 
addition to the COTAS main page and should help to eliminate confusion. However, it appears 
there is a minor and easily remedied error in the breakdown of percentages that indicate which 
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color an inmate or facility is sorted into. For example, green is categorized as 0-40.00% and the 
next category (yellow) begins at 41.00. Therefore if a facility or inmate has a score that falls 
between 40.00 and 41.00, they would not be accounted for in the color coding. When this issue 
was brought to the attention of Marien Dimacali, she looked back through the code and realized 
that the threshold categories are incorrect in the threshold color code tab. They are supposed to 
read as follows: 

- Green: 0.00% - 39.99% 

- Yellow: 40.00% - 75.00% 

- Red: 75.01% - 100% 

Therefore, the percentages need to be changed in the Threshold Color Coding tab, to indicate 
the proper breakdown of categories. 

Recommendation #2: DOC should make appropriate training opportunities available. 
Many respondents reported that they had never been properly trained and indicated that 
they would like to be trained to facilitate their ability to navigate COTAS. Perhaps 
voluntary training sessions could be offered at each parent facility along with the 
dissemination of a new, updated user’s manual. 

To date, there has been no training sessions made available to DOC staff. The user’s 
manual is located on the COTAS home page, as well as the quick threshold color coding tab 
within COTAS, on the RegionDashboard. Gail Denson has informed me that there is a central 
help desk for all DOC technical issues. Those experiencing problems with COTAS are re-routed 
to Gail’s department, whereupon the issue is addressed. This help desk method is the same one 
that was being used during the original validation (201 1). Additionally there is a help icon within 
COTAS that takes you to a detailed list of helpful tutorials, for example: exporting reports, report 
manager help. . .etc. There is also a search bar that allows for shortcuts to different screens. 

Lastly, on the bottom of the COTAS homepage there is a blurb that says “this website is 
maintained by Adam Hume.” If you click on his name, it allows you to email him directly if 
needs be. At this point it appears that training sessions are not be needed. The user’s manual and 
the threshold color code tab within COTAS have reconciled many of the issues identified by the 
users during the original validation. However the users were not surveyed during the re- 
validation so it cannot be determined if the need for additional training still exists. 

Recommendation #3 : Prior to updating the manual and developing training, DOC should 
coalesce on the primary purposed of COTAS, the desired uses/functions, and designate 
staff to provide “help desk” type assistance for users. Also, the DOC should monitor the 
usage/traffic, track error reports, and inaccurate data reports, and periodically solicit 
feedback and assessments from COTAS users in the field. 

The first page in the user’s manual clearly and succinctly explains the primary purpose of 
COTAS; “to identify issues before they develop.” It gives a brief introduction to the system, 


6 


This document is a research report submitted to the U.S. Department of Justice. This report has not 
been published by the Department. Opinions or points of view expressed are those of the author(s) 
and do not necessarily reflect the official position or policies of the U.S. Department of Justice. 



states its primary purpose and, explains how the system is structured. The user’s manual also 
highlights the desired uses and functions of COTAS (historical data and predictive data), as well 
as explaining the primary screens and functions DOC staff members should use COTAS based 
on job title. For example, there are sections specifically for Chief Security Officer that explains 
the ways in which this person would use COTAS. 

In terms of providing help desk assistance, monitoring the usage/traffic, and tracking 
error reports and inaccurate data reports, there is a system in place for all DOC employees to deal 
with technological problems. They call the central help desk and a work order is created and then 
disseminated to the appropriate department. In the case of COTAS problems, once a COTAS 
related work order is created, it is sent to Adam Hume, who is primarily responsible for COTAS 
related issues. Additionally Tommy Tucker (Database Administrator) runs usage reports 
monthly. At this time, the DOC is not periodically soliciting feedback and assessments from 
COTAS users into the field. 

Recommendation #4: As is common with many websites, there were occurrences of 
selecting links and being directed to a website that was not functioning properly. The non- 
functioning or broken links were not systematic; they were randomly dispersed throughout 
the COTAS screens. 

For the most part, the DOC has addressed this recommendation. The occurrence of 
broken links is not nearly as prevalent as it was during the original validation. In the original 
validation, roughly 25% of the links would either not work or direct you to the wrong website. 
However, during the re-validation there was only one incident of links not working as they 
should. The words are out of order in the statement of facts for all incidents (i.e. disciplinary 
reports, grievances, etc). For example the beginning is always located at the end, so the first few 
sentences are really the middle of the statement of facts. Marien Dimacali and Gail Denson have 
been made aware of this problem and Marien is currently working on fixing it. 

Recommendation #5: Occasionally the numbers in the columns did not add up accurately 
on the side of the RegionDashboard screen. This indicates to project staff that certain 
events were either not being counted or were being counted more than once. For example, 
an escape was listed as an escape and a non-violent disciplinary referral. 

During the re-validation, it was determined that this never was an issue. At the outset, it 
appears that the same incident is being counted twice, when in fact it is more likely the result of 
the inmate being charged with more than one disciplinary infraction. Instead of reporting by 
incident, they are reporting by disciplinary infraction received by each inmate. Additionally it is 
important to note that each employee is required to go through their own process each time an 
incident occurs. For example, if an inmate escapes it is entered in the Offender Based 
Information System (OBIS) as an escape. Once they are returned to the facility and receives a 
disciplinary infraction for escaping, the officer who gives them the disciplinary infraction enters 
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that information into OBIS. This makes it seem like the same incident is being counted twice 
when in fact the escape is counted as well as the disciplinary infraction received by the inmate. 
The same process is at work when it appears that some of the disciplinary infractions from the 
same incident are counted as violent and some are counted as non-violent. Each individual who 
was involved in the incident and received a disciplinary infraction as a result is entered into 
OBIS. So if there was an incident and some inmates were involved violently and some were 
involved non- violently, it appears the same incident is counted as both violent and non-violent. 
The fact is that some inmates were given violent disciplinary infractions and some were given 
non-violent disciplinary infractions, depending on the part they played in the incident. 

Recommendation #6: There are several institutions listed in COTAS that are no longer 
operational. There are several institutions that had been shut down yet they remained on 
the list in the COTAS interface. This may lead to the miscalculation of numerous variables. 
For example, it appears that each inmate is a member of their own gang when in reality it 
is zero inmates in zero gangs which yields a percentage of 100 for gang members in that 
facility. As a result, these facilities appear to be the most dangerous when in reality they are 
no longer in existence. 

For the most part, this recommendation has been addressed. Although the recent region 
switch (four regions became three and some facilities switched regions) caused some of the 
institutions to be listed in the wrong regions. For example: 

Fisted as Region 1 in OBIS and Region 2 in COTAS 
o Taylor Correctional Institute and Annex 

Fisted as Region 2 in OBIS and Region 3 in COTAS 

o Fowell Correctional Institute and Annex, Marion Correctional Institute and Work 
Camp, Putnam Correctional Institution, Tomoka Correctional Institution and 
Work Camp, Reality House, Daytona Work Release Center, Santa Fe Work 
Release Center, and Gainesville Work Camp 

Marien Dimacali and Gail Denson are aware of this problem and Marien is currently 
working on correcting this issue. 

There is an odd problem with the facility predictor in that it is predicting that institutions 
with only one or a few gang members have the highest probability for violence, when in fact the 
institutions with numerous gang members should have the highest probability for violence. 
Marien was notified of this issue via email with specific examples on September 6, 2013 and on 
September 9, 2013 the applicable facilities had inexplicably changed from a high risk of violence 
to a low risk of violence. For example: 

On 9/6 Tarpon Springs WRC had the highest probability of violence in Region 3 (with a 

GangRatePrediction of 0.929535155265153) even though they only had one gang (East 
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Side Bloods) with one member (James Torrey). However on 9/9 it is not ranked with the 
highest probability (with a GangRatePrediction of 0.471275428892822). 

On 9/6 Big Pine Key Road Prison was ranked second for the probability of a violent 
offense (with a GangRatePrediction of 0.857191088222281) even though they had no 
gangs with no members. However by 9/9 the GangRatePrediction had gone down to 
0.471275428892822. 

For both of these oddities, Marien could not explain the sudden switch. She is currently 
working on this issue. 

Recommendation #7: DOC should thoroughly review the interface, re-pilot test it, and 
revise the code to correct these abnormalities. Further it would be beneficial if DOC 
routinely or periodically sought input and feedback from users regarding improvements to 
the interface (e.g. additional screens, different layout, different presentation, options for 
customizing a screen, creating/saving reports). 

The majority of the issues pointed out during the original validation were reviewed and 
the code was revised and fixed. Even though another pilot test was recommended, this was not 
implemented. However this may no longer be necessary considering that the majority of the 
problems were addressed. The few problems that remain do not warrant a re -pilot test given that 
since the original validation the DOC has not solicited any feedback from the users in terms of 
the improvement of the interface. Additionally there are several ways for users to submit ideas 
for improvement, should they feel the need to do so, such as emailing Adam Hume directly, or 
calling the central help desk. 

Recommendation #8: Remove the color coding on the number of events at the regional level 
presented on the gauges on the top of the RegionGauges screen. The color coding that 
ranks the number of facilities with at least one violent event displayed as red (top of the 
RegionGauges screen), does not provide any interpretable information. Indeed, given that 
the regions vary in terms of the number of facilities, it may not make sense to apply a 
uniform metric based on count of facilities. Providing facility administrators with detailed 
statistics regarding events (both violent and non-violent) that have occurred within their 
facility is one of the major strengths of COTAS. According to the COTAS user’s survey, 
most administrators found this information to be quite helpful. However, it would be 
beneficial to provide the user with a detailed explanation of the color-coding systems and 
thresholds both in the user guide and on the COTAS interface. Additionally, thresholds 
should be updated on a regular basis to reflect current trends in the entire DOC system. 

In terms of removing the color coding on the number of events at the regional level 
(RegionGauge screen), this recommendation has not been addressed for legitimate reasons. 
Regions still vary in the number and types of facilities so applying a uniform count is not very 
interpretable. The color coding on the facilities with violent alerts is the same for all regions. The 
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ranges for color coding are 0-7 violent alerts put the facility in Green, 7-18 violent alerts put the 
facility in Yellow, and 18-25 violent alerts put the facility in Red. However, these ranges are not 
mentioned in the user’s manual or the threshold color coding tab. It would be beneficial to make 
these ranges available to the COTAS users by putting them in the user’s manual or the threshold 
color code tab. Additionally, the DOC might want to consider changing the ranges so that the 
categories do not overlap. For example, a facility with 7 violent alerts could be placed in green or 
yellow, and similarly a facility with 18 violent alerts could be placed in yellow or red. 

As explained in Recommendation #1, the user’s manual and the threshold color coding 
tab within COTAS explain the color coding and thresholds in a very comprehensive and 
effective manner. There is no mention in the user’s manual or the threshold color coding tab as to 
whether or not the thresholds are updated on a regular basis to reflect current trends in the entire 
DOC system. Instead the thresholds are separated by facility type (institution, work camp, work 
release center, etc). 

Recommendation #9: Provide users with explanations of the color coding systems and 
thresholds for inmates involvement in events over the past 30 days. 

This recommendation has been addressed. The updated user’s manual covers this 
extensively and the threshold color coding tab at the bottom of the COTAS welcome page gives 
a quick overview for the users once they have opened COTAS, allowing them a quick refresher 
without having to close COTAS and open the user’s manual. 

Recommendation #10: Update color coding thresholds for inmates’ involvement in events 
over the past 30 days at the facility level to reflect current data trends. 

In the original validation, the color coding for the facility gauges was based on the 
percentile rank that the number of violent event falls into relative to historical data for similar 
facilities, which lead to a considerable amount of confusion amongst the users of COTAS. It 
appears that this is still the case. Although once the thresholds had been clarified and the 
updated user’s manual disseminated, the use of the historical data for similar facilities is not so 
confusing. The users are able to see the events for the past 30 days in several different ways (raw 
count, graph), therefore the use of percentile rank relative to historical data for similar facilities 
is not so confusing and allows the users to view each facility compared to similar facilities 
historically, which could be a useful tool. 

Recommendation #11: Provide color coding of individual inmates on the 
PredictorSummaryByDorm screen. Inmates are assigned an inmate risk score from the 
COTAS algorithms and the scores are aggregated together into color coded categories at 
the facility and regional levels. It may be helpful to users to view these color categories 
when looking at a list of inmates or an inmate profile. 
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There is no color coding for the inmates in the PredictorS ummaryByDorm. It is listed by 
inmate, in descending order of their risk factor score. The user can click on the risk score and 
dial down to review the InmateSearch sheet. This lists the inmate’s DC information, their risk 
factor score, information on the prison in which they currently reside, as well as a history of their 
events for the last 30 days. 

Recommendation #12; Present data trends as changes in the rate of inmates involved in 
events, not the number of inmates involved in events (on RegionGraph and EventGraph 
screens). Changes in facilities’ population size may also increase or decrease the number of 
inmates involved in violent events. For example, a rate might be the number of inmates 
involved in violent events per 1,000 inmates. 

This recommendation has been addressed with the implementation of the data trends for 
both the RegionGraph and the EventGraph both show rate counts. However, the user’s manual 
indicates that they are incident counts, not rate counts. Therefore the user’s manual needs to be 
updated to reflect this change. At this time, Marien is aware of this issue and is working to 
resolve this issue by fixing the user’s manual. 

Recommendation #13: BRDA should be more actively involved in the creation and review 
of statistical modeling techniques employed by future versions of COTAS and, therefore, 
more involved with overseeing the work of subcontractor, Idea. 

According to Gail Denson, John Marsh, from Idea and David Ensley, the Director of the 
BRDA, worked very closely together both before and after the original validation. During the 
original validation, it was not apparent the extent to which they worked together, as their work 
was halted for the validation to take place. After the validation was submitted, John and David 
worked together extensively to address the recommendations. It appears that this 
recommendation has been addressed. 

Recommendation #14: Create a codebook or data dictionary of variables in the model and 
data warehouse and provide a clear description of the predictive model, including 
coefficients, measures of goodness of fit, and diagnostics. Additional problems with the 
COTAS algorithm may have been identified if contractors from Idea had provided clear 
documentation regarding model selection, algorithm diagnostics, and a codebook or data 
dictionary. 

This recommendation has been addressed. The data dictionary was completed on 
September 16, 2013. 

Recommendation #15: Use the actual dates of events and have the Microsoft Logistic 
Regression Algorithm model the predicted probability of inmates’ involvement in violent 
events within the following 30 days, and update the existing algorithm to reflect these 
changes. Many of the problems with the COTAS predictive models result from the fact that 


This document is a research report submitted to the U.S. Department of Justice. This report has not 
been published by the Department. Opinions or points of view expressed are those of the author(s) 
and do not necessarily reflect the official position or policies of the U.S. Department of Justice. 



the dates of events are changed from the actual date to the first day of the month . As a 
result, the time-order distinction is lost distinguishing events (e.g., inability to discern time 
order for a bed change to solitary confinement and a violent event). Further, there is a loss 
of data when the last day of the month falls on a weekend. 

This recommendation has been addressed. The queries and data used by the Microsoft 
Logistic Regression Algorithm (specifically DIM_INMATEPREDICTIONS) are using actual 
dates for events. 

Recommendation #16: Change the training data-set from static to dynamic with a “moving 
wall” of the previous 12 months of inmate data. COTAS should take full advantage of the 
Microsoft Logistic Regression Algorithm by changing the training data-set from static to 
dynamic and allowing the Algorithm to adjust to long-term changes in data or data trends, 
which may affect the impact of predictor variables on the outcomes. 

Based on interviews with DOC staff, it was determined that the DOC’s Bureau of 
Research and Data Analysis (BRDA) disagreed with this recommendation. The BRDA reasoning 
as provided by David Ensley, Director is shown below: 

“Allowing the system to self -modify would open up the potential of incremental system-wide 
changes that may go unnoticed. By having to "manually" modify the underlying models, more 
control over the modeling process is realized and these system-wide changes could be noted and 
analyzed. If there had been any of these major system-wide shifts that would warrant updating 
the underlying models, then it would have been apparent to the users of COTAS (when their 
entire facility showed up as high risk, for example). To date, BRDA has not received any 
feedback to this effect. ” 

Based on this explanation and based on industry best practice concerning baseline 
determinations, this recommendation has been addressed. 

Recommendation #17; Remediate deficiencies in the documentation of system 
requirements and design rationale, data warehouse design, predictive models, data 
visualization controls. 

This recommendation has been addressed. The COTAS Tutorial (User Guide), Technical 
Documentation, and Installation Manual were reviewed. Based on a comparison between the 
documentation and the database packages, ERD, and COTAS software, the documentation is up 
to date and reasonably comprehensive. Care should be taken to ensure that this documentation is 
maintained in concert with any future changes to the system. 

Recommendation #18: Re-design of the data warehouse dimensions and facts. 

The ERD was reviewed and the primary issues raised by the initial validation have been 
addressed. In particular, the use of NULL keys has been eliminated from the FACT_EVENT 
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table. These keys have been replaced by data fields. While certain issues may remain with slowly 
changing dimension, the load procedures document these risks and address them through data 
cleansing. While review and update of the data model should be an on-going task this 
recommendation has been addressed. 

Recommendation #19: Implement data cleansing in the extract-transform-load (ETL) 
process from the staging area to the data warehouse. 

During the re-validation, it was determined that the original recommendation may not 
have been necessary. This reviewer has examined the ETL packages used to load data from 
OBIS into STAGING and from STAGING into COTAS and found no issues with a lack of ‘data 
cleansing’. 

Recommendation #20: Perform formal testing of the ETL process and the reporting system 
with industry-standard documentation (plans, procedures and results). 

After request, the original vendor was unable to provide any documentation of formal 
testing that occurred. DOC Staff stated during interviews that testing was performed and the 
system was found satisfactory. It should be noted that the system has been in production for 
several years. While this recommendation cannot be considered fully addressed, it is the opinion 
of this reviewer that there is not a significant cost/benefit to conducting this testing at this late 
date for the purpose of generating the documentation. 

Recommendation #21: Future programming and coding work on COTAS should be done 
by contractors with extensive experience in data warehouse design and business 
intelligence. 

A review and evaluation of the curricula vitae of the contractors working on the COTAS 
system found no deficiencies in relevant experience. Therefore, this recommendation has been 
addressed. 


Summary 

COTAS has been validated through a joint effort by FSU’s College of Criminology and 
Criminal Justice, Center for Criminology and Public Policy Research and Infinity Software. A 
summary of the validation is listed below. 

Recommendation #1: This recommendation has been validated. The updated user’s 
manual and the threshold color code tab have been able to address the source of the 
majority of the confusion amongst the users of COTAS. The only issue that remains is to 
fix the color coding breakdown in the Threshold Color Coding Tab within COTAS to 
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reflect the proper breakdown of the percentages that place facilities and inmates into their 
respective color code. 


Recommendation #2: This recommendation has been validated. The updated user’s 
manual and the threshold color code tab address many of the issues identified in the 
original validation. Even though the DOC did not offer training sessions, it appears that 
between the helpful additions to COTAS (the user’s manual, the threshold color code tab 
within COTAS, the help desk icon, the search bar, Adam Hume’s email link, and the pre- 
existing DOC help desk system) the training sessions may not be needed. Although the 
users were not surveyed during the re-validation so we cannot say for sure if the need still 
exists. 


Recommendation #3: This recommendation has been validated. The user’s manual 
addresses the primary purpose of COTAS and the desired uses/functions. The help desk 
system that was in place during the original validation is still being used and appears to 
be quite effective. The Database Administrator, Tommy Tucker, runs usage reports. At 
this point, the DOC is not soliciting feedback and assessments from COTAS to users in 
the field. 

Recommendation #4: This recommendation has been validated. The broken links have 
been fixed. The only issue that remains is the order of the words for all of the statement 
of facts. In every case, the beginning is always located at the end. Gail Denson and 
Marien Dimacali have been made aware of the issue and Marien is currently working on 
it. 

Recommendation #5: This recommendation has been validated. What appears to be 
double counting of an incident is in fact the occurrence of more than one disciplinary 
infraction received by an inmate for the same incident. 

Recommendation #6: This recommendation has been validated. Facilities that are no 
longer in existence are no longer listed on COTAS. Although two problems still exist. 
Recently the DOC has changed from four regions to three, causing several institutions to 
be switched into a new region. At this point, several facilities are still listed in the 
incorrect region in COTAS. Additionally the facility predictor appears to be rapidly 
changing the GangPredictionRate of some facilities, thereby causing these facilities to be 
placed in a the red category one day and the green category a few days later. Marien 
Dimacali is aware of this issue and is working to fix it. 
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Recommendation #7 : This recommendation has been validated. The majority of the 
issues pointed out during the original validation were revised and fixed. Although the 
DOC did not re-pilot test COTAS, it appears this is no longer necessary since the 
majority of the problems were addressed. Additionally, the DOC does not solicit any 
feedback from users as was suggested during the original validation. 

Recommendation #8: This recommendation has been validated. The DOC only partially 
addressed this recommendation. The user’s manual and the threshold color code tab 
comprehensively explained the color coding and thresholds. But the DOC did not remove 
the color coding of the number of events at the region level (RegionGauge), as was 
recommended during the original validation. Additionally the DOC did not mention in 
the user’s manual or the threshold color coding tab whether or not the thresholds are 
updated on a regular basis. Lastly, the threshold categories that determine what color a 
facility is coded overlap. A facility that has 7 violent alerts could be placed in green or 
yellow, and a facility with 18 violent alerts could be placed in yellow or red. Marien 
Dimacali is aware of this issue. 


Recommendation #9: This recommendation has been validated. The color coding system 
has been thoroughly addressed by the updated user’s manual and the threshold color 
coding tab. 

Recommendation #10: This recommendation has been validated. Although the DOC did 
not change COTAS to reflect this recommendation, it became unnecessary once the 
thresholds had been thoroughly explained, eliminating the confusion surrounding the 
historical data. 

Recommendation #11: This recommendation has been validated. At this point the DOC 
has not addressed this recommendation. There is no color coding for the inmates in the 
PredictorS ummaryB yD orm. 

Recommendation #12: This recommendation has been validated. The DOC changed the 
RegionGraph and the EventGraph to present data trends as changes in rates of inmates 
involved in events, as opposed to event counts. Although the user’s manual indicates that 
it is event counts, not rate counts. Marien Dimacali has been made aware of this problem 
and is working to fix it. 

Recommendation #13: This recommendation has been validated. BRDA is actively 
involved in the statistical modeling techniques. David Ensley, the Director of BRDA, 
worked very closely with John Marsh from Idea. 
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Recommendation #14: This recommendation has been validated. The data dictionary was 
completed on September 16, 2013. 

Recommendation #15: This recommendation has been validated. The algorithms used to 
predict inmate involvement in violent events have been properly updated. 

Recommendation #16: This recommendation has been validated. The BRDA disagreed 
with the recommendation and this reviewer agrees with that explanation 

Recommendation #17: This recommendation has been validated. The documentation was 
reviewed and found to be sufficient and up-to-date. At this point, DOC is in the process 
of creating a Data Dictionary to complete the technical documentation. 

Recommendation #18: This recommendation has been validated. The ERD was reviewed 
and the technical deficiencies in warehouse design have been addressed. 

Recommendation #19: This recommendation has been validated. At this point, the DOC 
is still in the process of creating one. 

Recommendation #20: This recommendation has not been validated. While DOC staff 
agree that testing occurred, documentation was either not created or not retained by DOC 
or the vendor. At this stage, this reviewer does not recommend retesting. 

Recommendation #21 : This recommendation has been validated. While this reviewer 
found this recommendation to be unusual and vague, a review of the resumes and 
relevant information for the vendor team found no issues. 
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